Export And Commit Table Schemas

📄 <ENV> - [BigQuery] Export and Commit Table Schemas

This GitHub Actions workflow automates the extraction of BigQuery table schemas from the bqDataLake dataset and commits them to the repository. It helps ensure that your schema definitions remain version-controlled and in sync with BigQuery.

---

📌 Workflow Trigger

This workflow is triggered manually via the GitHub actions in function repo.

---

💠 Environment Variables

Variable	Description
GCP_PROJECT_ID	GCP Project ID (e.g., biddirect-2)
DATASET_ID	BigQuery dataset ID (e.g., bqDataLake)

---

🧱️ Workflow Job: export-and-commit-bigquery-schemas

Step	Description
Checkout repository	Clones the GitHub repository to the runner.
Authenticate with GCP	Authenticates using a service account (<ENV>_BIGQUERY_ADMIN_SERVICE_ACCOUNT) stored in GitHub Secrets.
Set up Cloud SDK	Installs and configures the Google Cloud SDK CLI.
Create output directory	Creates the directory to store schema JSON files.
Get all schemas from BigQuery	Iterates over all tables in the dataset (excluding _changelog and non-table types), exports schema definitions, and saves them to local files.
Commit and Push Schema Files	Stages and commits any updated schema files to the repository.

---

🧹 Export Logic Summary

• Skips tables ending with _changelog

• Skips views or other non-TABLE resources

• Removes _latest suffix from table names when naming schema files

• Stores output in:

  functions/src/bqDataLake/definitions/tableSchemas/

Example:

Table: accounts_latest → File: accounts.json

---

🔐 Required Secrets

Secret Name	Description
<ENV>_BIGQUERY_ADMIN_SERVICE_ACCOUNT	Service account key JSON for GCP auth.

---

✅ Example Usage

To run this workflow:

1. Go to the Actions tab in your GitHub repository.
2. Select the workflow <ENV> - [BigQuery] Export and Commit Table Schemas.
3. Click "Run workflow".

This will:

• Export schemas for all physical _latest tables in the bqDataLake dataset
• Save each schema in JSON format to the repo
• Automatically commit and push the updated files to the branch from which the workflow was executed

---

This workflow ensures that schema changes are tracked alongside application code.